Papers with Natural Language Generation
Catch Me If You GPT: Tutorial on Deepfake Texts (2024.naacl-tutorials)
Copied to clipboard
| Challenge: | In recent years, natural language generation (NLG) techniques have advanced, but pose new security risks . this tutorial will be 3 hours long with a mix of lecture and hands-on examples for interactive audience participation. |
| Approach: | They present a tutorial on the security of natural language generation (NLG) they review the latest literature on the detection and obfuscation of deepfake text authorships . |
| Outcome: | This tutorial reviews the latest literature on the detection and obfuscation of deepfake text authorships. |
Towards Generation and Recognition of Humorous Texts in Portuguese (2023.eacl-srw)
Copied to clipboard
| Challenge: | This PhD thesis focuses on the automatic generation and recognition of verbal punning humor in Portuguese. |
| Approach: | They propose to combine natural language generation and cognitive processing to generate and recognize verbal humor in Portuguese. |
| Outcome: | The proposed methods aim to generate and recognize humor in Portuguese, an underdeveloped language compared to English. |
A Tutorial on Evaluation Metrics used in Natural Language Generation (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
| Approach: | This tutorial presents the evolution of automatic evaluation metrics to their current state . it aims to assess the extent of scientific progress made and identify areas/components that need improvement . |
| Outcome: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
Natural Language Generation: Recently Learned Lessons, Directions for Semantic Representation-based Approaches, and the Case of Brazilian Portuguese Language (P19-2)
Copied to clipboard
| Challenge: | Natural Language Generation (NLG) is a promising area in Natural Language Processing (NLP) . |
| Approach: | They present a review of the literature on Natural Language Generation in Brazilian Portuguese. |
| Outcome: | The proposed approaches are based on the Abstract Meaning Representation formalism and have potential future directions. |
Generate, Prune, Select: A Pipeline for Counterspeech Generation against Online Hate Speech (2021.findings-acl)
Copied to clipboard
| Challenge: | Off-the-shelf methods to generate hate speech are limited in that they generate repetitive and safe responses regardless of the hate speech. |
| Approach: | They propose a three-module pipeline approach to generate diverse and relevant counterspeech . they first generate various counterspeak candidates by a generative model, then filter ungrammatical ones using a BERT model . |
| Outcome: | The proposed pipeline generates diverse and relevant counterspeech responses on three datasets. |
Back-Translation as Strategy to Tackle the Lack of Corpus in Natural Language Generation from Semantic Representations (D19-63)
Copied to clipboard
| Challenge: | Abstract Meaning Representation and Brazilian Portuguese (BP) are selected as semantic representation and language, respectively. |
| Approach: | They propose to use Brazilian Portuguese and Abstract Meaning Representation as semantic representations for NLG. |
| Outcome: | The proposed methods were evaluated on two datasets (one automatically generated and another human-generated) to compare the performance in a real context. |
Exploring Chunk Based Templates for Generating a subset of English Text (P18-3)
Copied to clipboard
| Challenge: | In this paper, we address the problem of the automatic generation of natural language text representative of an input non-linguistic collection of knowledge. |
| Approach: | They propose a fitness function which is used to generate grammatical sentences . they use an evolutionary algorithm to model the fitness of the sentence . |
| Outcome: | The proposed approach uses an evolutionary algorithm to arrive at a potentially grammatical sentence which satisfies the input constraints. |
Few-shot Natural Language Generation for Task-Oriented Dialog (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for NLG depend on heavily annotated data, which is infeasible for new domains. |
| Approach: | They propose a system that converts a dialog act into a response in natural language . they propose 'nuclear language generation' to simulate a few-shot learning setting . |
| Outcome: | The proposed model outperforms existing methods on a large set of annotated datasets. |
Pre-train and Plug-in: Flexible Conditional Text Generation with Variational Auto-Encoders (2020.acl-main)
Copied to clipboard
| Challenge: | Existing conditional generation models cannot handle emerging conditions due to their joint end-to-end learning fashion. |
| Approach: | They propose a framework for conditional text generation that decouples the text generation module from the condition representation module to allow "one-to-many" conditional generation. |
| Outcome: | The proposed framework decouples the text generation module from the condition representation module to allow “one-to-many” conditional generation. |
An Error Analysis Framework for Shallow Surface Realization (2021.tacl-1)
Copied to clipboard
| Challenge: | BLEU and METEOR metrics fail to provide information on which linguistic factors impact performance of natural language generation models. |
| Approach: | They propose a framework for error analysis which permits identifying which features of the input affect the models’ results. |
| Outcome: | The proposed framework improves the performance of 174 system runs submitted to the Multilingual SR shared tasks. |
Generate, Filter, and Rank: Grammaticality Classification for Production-Ready NLG Systems (N19-2)
Copied to clipboard
| Challenge: | Existing datasets for grammatical error correction don’t capture the distribution of errors that data-driven generators are likely to make. |
| Approach: | They propose a framework that allows candidates to be filtered and ranked to select the best response. |
| Outcome: | The proposed framework can be scaled with relatively low effort and achieve high precision with reasonable recall on a weather domain dataset. |
ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation (2021.acl-demo)
Copied to clipboard
Weizhen Qi, Yeyun Gong, Yu Yan, Can Xu, Bolun Yao, Bartuer Zhou, Biao Cheng, Daxin Jiang, Jiusheng Chen, Ruofei Zhang, Houqiang Li, Nan Duan
| Challenge: | Existing models for pre-training are not convenient for users to find and set them up. |
| Approach: | They propose to extend ProphetNet into other domains and languages by pre-training models . they pre-train a cross-lingual generation model ProphetNet-Multi and a Chinese generation model . |
| Outcome: | The proposed models achieve new state-of-the-art on 10 benchmarks. |
Automatic Unit Test Data Generation and Actor-Critic Reinforcement Learning for Code Synthesis (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large pre-trained language models have shown remarkable performance on various benchmarks, treating the problem of Code Generation in a fashion similar to Natural Language Generation, trained with a Language Modelling (LM) objective. |
| Approach: | They propose a method to automatically obtain function signatures and associated Unit Tests, suitable for RL training of Code Synthesis models. |
| Outcome: | The proposed model improves on a pre-trained model on various benchmarks and is more efficient than previous models. |
GLGE: A New General Language Generation Evaluation Benchmark (2021.findings-acl)
Copied to clipboard
Dayiheng Liu, Yu Yan, Yeyun Gong, Weizhen Qi, Hang Zhang, Jian Jiao, Weizhu Chen, Jie Fu, Linjun Shou, Ming Gong, Pengcheng Wang, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, Ruofei Zhang, Winnie Wu, Ming Zhou, Nan Duan
| Challenge: | Multi-task benchmarks focus on a range of Natural Language Understanding (NLU) tasks without considering the Natural Language Generation (NLG) models. |
| Approach: | They propose a multi-task benchmark for evaluating the generalization capabilities of NLG models across eight language generation tasks. |
| Outcome: | The proposed benchmarks are based on GLUE and Su-perGLUE for English and several other languages. |
Fact Checking Machine Generated Text with Dependency Trees (2022.emnlp-industry)
Copied to clipboard
| Challenge: | Recent work has noted the benefits of natural language text generated by NLG systems over fixed templates. |
| Approach: | They propose a method that checks factuality of input text based on structured knowledge patterns and dependency relations with respect to the input text. |
| Outcome: | The proposed technique outperforms state-of-the-art techniques in this special, but important case. |
DATScore: Evaluating Translation with Data Augmented Translations (2023.findings-eacl)
Copied to clipboard
| Challenge: | Experimental results show that DATScore correlates better with human meta-evaluations than the other recent state-of-the-art metrics. |
| Approach: | They propose to use data augmented translations to improve the evaluation of machine translations by using two new scoring strategies. |
| Outcome: | The proposed metric improves on 3 NLG tasks other than translation. |
NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist (2023.acl-long)
Copied to clipboard
| Challenge: | a systematic review of automatic evaluation metrics for Natural Language Generation (NLG) shows that task-agnostic metrics have a weak correlation with human . |
| Approach: | They propose a framework to assess the effectiveness of automatic metrics in three NLG tasks . they propose task-agnostic and human-aligned metrics to be used for evaluation . |
| Outcome: | The proposed framework provides access to the evaluation tools for three NLG tasks. |
AugNLG: Few-shot Natural Language Generation using Self-trained Data Augmentation (2021.acl-long)
Copied to clipboard
| Challenge: | Large-scale conversational systems typically generate unnatural, robotic responses using template-based approaches. |
| Approach: | They propose a data augmentation approach that combines a self-trained neural retrieval model with a few-shot learned NLU model to automatically create MR-to-Text data from open-domain texts. |
| Outcome: | The proposed approach outperforms the state-of-the-art methods on the FewshotWOZ data in both BLEU and Slot Error Rate. |
Adversarial Domain Adaptation for Variational Neural Language Generation in Dialogue Systems (C18-1)
Copied to clipboard
| Challenge: | Existing methods to adapt to domains have shown promising results in how to reuse data in a domain-scalable framework efficiently. |
| Approach: | They propose an adversarial training procedure to train a Variational encoder-decoder based language generator via multiple adaptation steps. |
| Outcome: | The proposed method can adapt to a related domain using only a small amount of in-domain data. |
Multi-task Learning for Natural Language Generation in Task-Oriented Dialogue (D19-1)
Copied to clipboard
| Challenge: | Existing methods to generate natural language for task-oriented dialogues lack naturalness and variation in language. |
| Approach: | They propose a multi-task learning framework for natural language generation that explicitly targets for naturalness in generated responses via an unconditioned language model. |
| Outcome: | The proposed framework outperforms existing models across multiple datasets in the study of natural language generation. |
Towards a Unified Multi-Dimensional Evaluator for Text Generation (2022.emnlp-main)
Copied to clipboard
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, Jiawei Han
| Challenge: | Existing evaluation frameworks for natural language generation are dominated by similarity-based metrics. |
| Approach: | They propose a multi-dimensional evaluator for natural language generation that integrates multiple dimensions into one evaluer. |
| Outcome: | The proposed evaluator improves on three typical NLG tasks and improves with external knowledge. |
Logic2Text: High-Fidelity Natural Language Generation from Logical Forms (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies on Natural Language Generation (NLG) from structured data focus on surface descriptions of simple record sequences, for example, attribute-value pairs of fixed or very limited schema. |
| Approach: | They propose to use a large-scale dataset to generate NLG from logical forms to obtain controllable and faithful generations from structured data. |
| Outcome: | The proposed model can describe interesting facts from logical inferences across records, but it is difficult to produce such fidelity. |
A Study of Automatic Metrics for the Evaluation of Natural Language Explanations (2021.eacl-main)
Copied to clipboard
| Challenge: | a lack of transparency is a key issue for robotics and AI. |
| Approach: | They propose to map existing automatic evaluation methods for natural language generation onto explanations. |
| Outcome: | The proposed model shows that embedding-based evaluation methods have higher correlations with human ratings than word-overlap metrics. |
Evaluating the Effect of Retrieval Augmentation on Social Biases (2026.eacl-long)
Copied to clipboard
| Challenge: | RAG is a popular method for injecting up-to-date knowledge into LLMs. |
| Approach: | They examine how RAG modulates social biases across three languages and four categories . they find that biased documents are amplified even when base LLM has low-level of intrinsic bias . |
| Outcome: | The proposed method can enhance factual accuracy but its effect on social biases is not well understood. |
Jailbreak Open-Sourced Large Language Models via Enforced Decoding (2024.acl-long)
Copied to clipboard
Hangfan Zhang, Zhimeng Guo, Huaisheng Zhu, Bochuan Cao, Lu Lin, Jinyuan Jia, Jinghui Chen, Dinghao Wu
| Challenge: | Existing studies show that Large Language Models can be misused to generate undesired content. |
| Approach: | They propose to use large language models to manipulate the generation process to generate undesired content without heavy computations or prompt designs. |
| Outcome: | The proposed method shows that open-sourced large language models could be misused to generate undesired content without heavy computations or prompt designs. |
CodePrompt: Task-Agnostic Prefix Tuning for Program and Language Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Prompt-tuning methods have been used to solve inefficient parameter update and storage issues in Natural Language Generation tasks. |
| Approach: | They propose a task-agnostic prompt tuning method that reflects the traits of PLM for program language. |
| Outcome: | The proposed method is effective in three PLG tasks, not only in the full-data setting but also in the low-resource setting and cross-domain setting. |
IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languages (2022.emnlp-main)
Copied to clipboard
Aman Kumar, Himani Shrotriya, Prachi Sahu, Amogh Mishra, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, Mitesh M. Khapra, Pratyush Kumar
| Challenge: | IndicNLG is a non-English language that is hampered by the scarcity of datasets. |
| Approach: | They propose to create a dataset for natural language generation for 11 Indic languages . they use a set of pre-trained models to train multilingual models . |
| Outcome: | The proposed datasets show that pre-trained models perform well in multilingual and monolingual tasks. |
Does Meta-learning Help mBERT for Few-shot Question Generation in a Cross-lingual Transfer Setting for Indic Languages? (2022.coling-1)
Copied to clipboard
Aniruddha Roy, Rupak Kumar Thakur, Isha Sharma, Ashim Gupta, Amrith Krishna, Sudeshna Sarkar, Pawan Goyal
| Challenge: | Existing approaches to few-shot Question Generation (QG) are limited and require manual annotation. |
| Approach: | They propose to use multilingual BERT to perform few-shot question generation with cross-lingual transfer. |
| Outcome: | The proposed model improves in few-shot QG and human evaluation confirms it. |
Intrinsic Task-based Evaluation for Referring Expression Generation (2024.acl-long)
Copied to clipboard
| Challenge: | Referring Expression Generation (REG) models generate referring expressions that refer to referents at different points in a discourse. |
| Approach: | They propose to use a purely ratings-based human evaluation to evaluate REG models by completing two meta-level tasks. |
| Outcome: | The proposed evaluation makes the models more reliable and discriminable, and improves the quality of the REs. |
Can Language Models Make Fun? A Case Study in Chinese Comical Crosstalk (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models have been widely used in NLP, but their social or cultural impact is under-explored. |
| Approach: | They build a dataset consisting of numerous **C**hinese **C*omical **C***rosstalk scripts, which is for a popular Chinese performing art called ‘Xiangsheng’ or ‘’ since 1800s. |
| Outcome: | The proposed approach can generate humor as humans do, but it is still in its infancy. |
Unsupervised Natural Language Generation with Denoising Autoencoders (D18-1)
Copied to clipboard
| Challenge: | Unsupervised approaches to generating text from structured data are costly to obtain and limited to a limited domain. |
| Approach: | They propose an unsupervised approach that learns its parameters without the slot pairs on target sequences only. |
| Outcome: | The proposed approach can generate sentences out of corrupted data without supervision . it can be used in question answering and dialog systems, the authors show . |
Defining and Detecting Vulnerability in Human Evaluation Guidelines: A Preliminary Study Towards Reliable NLG Evaluation (2024.naacl-long)
Copied to clipboard
| Challenge: | Unreliable evaluation guidelines can yield inaccurate assessment outcomes, potentially impeding the advancement of NLG in the right direction. |
| Approach: | They propose to collect annotated human evaluation guidelines and a method for detecting guideline vulnerabilities using Large Language Models. |
| Outcome: | The proposed dataset includes eight vulnerabilities and a method for detecting guideline vulnerabilities. |
DHP Benchmark: Are LLMs Good NLG Evaluators? (2025.findings-naacl)
Copied to clipboard
Yicheng Wang, Jiayi Yuan, Yu-Neng Chuang, Zhuoer Wang, Yingchi Liu, Mark Cusick, Param Kulkarni, Zhengping Ji, Yasser Ibrahim, Xia Hu
| Challenge: | Large Language Models (LLMs) are increasingly serving as evaluators in Natural Language Generation (NLG) tasks. |
| Approach: | They propose a framework that measures the discernment of Large Language Models (LLMs) across diverse NLG tasks. |
| Outcome: | The proposed framework provides quantitative discernment scores for LLMs across four NLG tasks. |
X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspects (2024.naacl-long)
Copied to clipboard
| Challenge: | X-Eval is a two-stage instruction tuning framework to evaluate text in both seen and unseen aspects customized by end users. |
| Approach: | They introduce a two-stage instruction tuning framework to evaluate text in both seen and unseen aspects customized by end users. |
| Outcome: | The proposed framework improves the model’s ability to follow evaluation instructions and enhances the learning stage to better assess text quality. |
Definite Description Lexical Choice: taking Speaker’s Personality into account (L18-1)
Copied to clipboard
| Challenge: | Referring Expression Generation (REG) lexical choice is the subtask that provides words to express an input meaning representation. |
| Approach: | They propose a personality-dependent lexical choice model for Referring Expression Generation (REG) that provides words to express a given input meaning representation. |
| Outcome: | The proposed model outperforms a standard lexicalisation model based on meaning-to-text mappings and personality information. |
Up-cycling Data for Natural Language Generation (L18-1)
Copied to clipboard
| Challenge: | Existing systems for creating adaptive texts from cultural heritage data require expert input . a number of research projects have focused on using NLG systems to create multilingual adaptive texts . |
| Approach: | They propose automatic processes which aim to reduce the need for expert input . they normalize the dates and names which occur in the data and link to the Semantic Web . |
| Outcome: | The proposed processes reduce the need for expert input during conversion and up-cycling process. |
Enhancing Task-Specific Distillation in Small Data Regimes through Language Generation (2022.coling-1)
Copied to clipboard
| Challenge: | Large-scale pretrained language models have led to significant improvements in Natural Language Processing, but they come at the cost of high computational and storage requirements. |
| Approach: | They propose to distill knowledge from larger models to smaller ones through pseudo-labels on task-specific datasets. |
| Outcome: | The proposed approach improves on the SST-2, MRPC, YELP-2, and TREC-6 datasets. |
Template Guided Text Generation for Task-Oriented Dialogue (2020.emnlp-main)
Copied to clipboard
| Challenge: | a new study examines the use of templates to generate natural language utterances for a large number of APIs. |
| Approach: | They propose a schema-guided approach which conditions the generation on a natural language schema. |
| Outcome: | The proposed method improves over strong baselines, is robust to out-of-domain inputs and shows improved sample efficiency. |
T-REx: A Large Scale Alignment of Natural Language with Knowledge Base Triples (L18-1)
Copied to clipboard
Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, Elena Simperl
| Challenge: | Existing datasets that provide alignments between natural language and knowledge bases (KB) triples are limited in size, lack coverage and are of unreported quality. |
| Approach: | They propose to build a large scale dataset of alignments between Wikipedia abstracts and Wikidata triples that is two orders of magnitude larger than the largest available alignments dataset. |
| Outcome: | The proposed dataset is two orders of magnitude larger than the largest available dataset and covers 2.5 times more predicates. |
Towards Pragmatic Production Strategies for Natural Language Generation Tasks (2022.emnlp-main)
Copied to clipboard
| Challenge: | Using language to communicate successfully requires effort. |
| Approach: | They propose a conceptual framework for the design of natural language generation systems that follow efficient and effective production strategies to achieve complex communicative goals. |
| Outcome: | The proposed framework is applied to visually grounded referential games and abstractive text summarisation tasks with real-world applications. |
Human-Machine Collaboration Approaches to Build a Dialogue Dataset for Hate Speech Countering (2022.emnlp-main)
Copied to clipboard
| Challenge: | a new approach to combat online hate speech is being proposed for NLG . existing methods to train NLG are limited to 2-turn interactions, while in real life, interactions can consist of multiple turns. |
| Approach: | They propose to combine human annotators with machine generated dialogues to create a dataset . DIALOCONAN is the first dataset comprising over 3000 fictitious multi-turn dialogues . |
| Outcome: | The proposed approach combines human experts over machine generated dialogues . it is the first dataset comprising over 3000 fictitious multi-turn dialogues between a hater and an NGO operator . |
Perturbation CheckLists for Evaluating NLG Evaluation Metrics (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics for natural language generation are inadequate . existing metrics are not robust against simple perturbations and disagree with scores assigned by humans to perturbed output. |
| Approach: | They propose to propose checks which perturb the output and target a specific criteria and then use them to refine their evaluation. |
| Outcome: | The proposed templates show that existing evaluation metrics are not robust against simple perturbations and disagree with human scores on the perturbed output. |
Best Student Forcing: A Simple Training Mechanism in Adversarial Language Generation (2020.lrec-1)
Copied to clipboard
| Challenge: | Language models trained with Maximum Likelihood Estimation (MLE) have been considered as a mainstream solution in Natural Language Generation (NLG) however, they are reportedly suffering from training instability and mode collapse, and therefore outperform conventional MLE models. |
| Approach: | They propose a method to improve Generative Adversarial Nets (GANs) using best student forcing and discriminators to increase training stability and sample diversity. |
| Outcome: | The proposed techniques outperform MLE models and outperformed existing approaches in terms of sample diversity and training stability. |
Exploring Transformer Text Generation for Medical Dataset Augmentation (2020.lrec-1)
Copied to clipboard
| Challenge: | Natural Language Processing (NLP) is a powerful tool to unlock the vast troves of unstructured data in clinical text. |
| Approach: | They propose a method for augmenting unstructured patient information to allow NLP model development on downstream clinically relevant tasks. |
| Outcome: | The proposed method beats baselines on a downstream classification task and can be used for NLP model development. |
AmbigNLG: Addressing Task Ambiguity in Instruction for NLG (2024.emnlp-main)
Copied to clipboard
| Challenge: | AmbigNLG is a novel task designed to tackle task ambiguity in instructions for NLG . ambiguous instructions often impede the performance of Large Language Models (LLMs) . |
| Approach: | They propose an ambiguity taxonomy that categorizes different types of instruction ambiguities and refines initial instructions with clearer specifications. |
| Outcome: | The proposed task improves alignment of generated text with user expectations, achieving 15.02-point increase in ROUGE scores. |
Active Learning for Natural Language Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to NLG are limited by the lack of annotated data. |
| Approach: | They propose to use active learning to reduce the cost of manual annotation to improve annotation efficiency by selecting the most informative examples to label. |
| Outcome: | The proposed approach surpasses baseline of random example selection in some cases but not in others. |
The ACL OCL Corpus: Advancing Open Science in Computational Linguistics (2023.emnlp-main)
Copied to clipboard
| Challenge: | ACL OCL is a scholarly corpus derived from the ACL Anthology . it provides metadata, PDF files, citation graphs and additional structured full texts . |
| Approach: | They present ACL OCL, a scholarly corpus derived from the ACL Anthology . it integrates metadata, PDF files, citation graphs and additional structured full texts . they highlight how it applies to observe trends in computational linguistics . |
| Outcome: | The ACL OCL spans seven decades and contains 73,285 papers . the scholarly corpus is based on the ACL Anthology and is available from HuggingFace . |
Contrastive Error Attribution for Finetuned Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for error tracing do not detect faithfulness errors in NLG datasets. |
| Approach: | They propose a framework to identify and remove low-quality training instances that lead to undesirable outputs. |
| Outcome: | The proposed method outperforms existing methods for detecting faithfulness errors in NLG datasets. |
Shape of Synth to Come: Why We Should Use Synthetic Data for English Surface Realization (2020.acl-main)
Copied to clipboard
| Challenge: | In the Surface Realization Shared Tasks of 2018 and 2019, there was little difference in absolute performance between systems trained with and without synthetic data. |
| Approach: | They propose to use synthetic data to explore approaches to surface realization from Universal-Dependency-like trees to surface strings for several languages. |
| Outcome: | The proposed method improves performance of a previously state-of-the-art system by 8 BLEU points over the previous system on the English dataset. |
Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics are conflated and can mislead models, resulting in downstream harms. |
| Approach: | They propose a framework for conceptualizing and evaluating the reliability and validity of evaluation metrics based on empirical data. |
| Outcome: | The proposed framework formalizes the source of measurement error and offers statistical tools for evaluating evaluation metrics based on empirical data. |
DSPM-NLG: A Dual Supervised Pre-trained Model for Few-shot Natural Language Generation in Task-oriented Dialogue System (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing models for few-shot natural language generation are based on a dual correlation between NLG and SLU from the perspective of probability. |
| Approach: | They propose a dual supervised pre-trained model to regularize the pre-training process . they use a probabilistic approach to learn the dual correlation between NLG and SLU . |
| Outcome: | The proposed model outperforms the previous state-of-the-art models on a few-shot dataset. |
A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training (2023.acl-long)
Copied to clipboard
| Challenge: | Modern Natural Language Generation models come with massive computational and storage requirements. |
| Approach: | They propose a method that applies word-level knowledge distillation to multiple PTs generated by both teacher and student. |
| Outcome: | The proposed techniques can be used to compress natural language models while preserving their performance. |
FOCUS: Forging Originality through Contrastive Use in Self-Plagiarism for Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to generate original text using pre-trained language models are problematic as they are trained on corpora constructed by human authors. |
| Approach: | They propose a unique “self-plagiarism” contrastive decoding strategy that modifies prompts in LLMs to develop an amateur model and a professional model. |
| Outcome: | The proposed method enables the development of an amateur model and a professional model while maintaining its standard language model status. |
What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability (2023.emnlp-main)
Copied to clipboard
| Challenge: | In Natural Language Generation tasks, multiple communicative goals are plausible and any goal can be put into words, or produced, in multiple ways. |
| Approach: | They characterise the extent to which human production varies lexically, syntactically, and semantically across four NLG tasks, connecting human production variability to aleatoric or data uncertainty. |
| Outcome: | The proposed model can be calibrated to human production variability using multiple samples and, when possible, multiple references. |
Leveraging Large Language Models for NLG Evaluation: Advances and Challenges (2024.emnlp-main)
Copied to clipboard
| Challenge: | introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. |
| Approach: | They propose a taxonomy for organizing existing LLM-based evaluation metrics and a structured framework to understand and compare them. |
| Outcome: | The proposed taxonomy offers a framework to understand and compare LLM-based evaluation methods. |
Reinforcement Replaces Supervision: Query focused Summarization using Deep Reinforcement Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Query-focused Summarization (QfS) is a system that generates summaries from document(s) based on a query. |
| Approach: | They propose a Query-focused Summarization approach that uses a generalization of Reinforcement Learning (RL) for Natural Language Generation and a better semantic similarity reward. |
| Outcome: | The proposed approach improves on the ROUGE-L metric and in a benchmark dataset. |
Evaluating Diversity in Automatic Poetry Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for creative text generation are not evaluated regarding how different generated poems are from existing training sets. |
| Approach: | They evaluate the diversity of automatically generated poetry by comparing distributions of generated poetry to distributions in human poetry along structural, lexical, semantic and stylistic dimensions. |
| Outcome: | The proposed model types show that style-conditioning and character-level modeling increases diversity across virtually all dimensions. |
Towards A “Novel” Benchmark: Evaluating Literary Fiction with Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) context windows have enabled them to process inputs over 100K tokens and generate outputs of up to 10K token. |
| Approach: | They propose a multi-level evaluation framework that incorporates ten metrics across the Macro, Meso, and Micro levels and an annotated fiction dataset. |
| Outcome: | The proposed framework incorporates ten metrics across the Macro, Meso, and Micro levels and is based on a human-human-AI dataset. |
Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Using large language models (LLMs) for evaluating natural language generation has gained traction . lm judges have low intra-rater reliability in their assigned scores, making it difficult to measure how good their judgments actually are. |
| Approach: | They show that large language models align more closely with human preferences than n-grams . they quantify this variance and compare them to other NLG tasks and benchmarks based on the results . |
| Outcome: | The proposed models align more closely with human preferences than n-gram or embedding-based metrics. |
Argument Summarization and its Evaluation in the Era of Large Language Models (2025.emnlp-main)
Copied to clipboard
Moritz Altemeyer, Steffen Eger, Johannes Daxenberger, Yanran Chen, Tim Altendorf, Philipp Cimiano, Benjamin Schiller
| Challenge: | Large Language Models (LLMs) have revolutionized various Natural Language Generation tasks, including Argument Summarization (ArgSum). |
| Approach: | They propose a prompt-based evaluation scheme and validate it through a human benchmark dataset. |
| Outcome: | The proposed evaluation scheme outperforms existing methods and is validated by a human benchmark dataset. |